The codes can be run using MATLAB. 
Run main1.m file. 
Change the system parameters as you need

ActionValue.m : Calculates action value function with predicted transition kernels 
OurPolicy1.m: Our Policy
random1.m: random policy
UCWhittle1.m: UCWhittle Policy
WhittleOracle1.m: Whittle Index Policy with with True Transition Kernels
WhittleIndex1.m: Calculates Whittle index with predicted transition kernels
WIQL1; WIQL policy